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ABSTRACT 

The overall risk for any spacecraft system is typically 
determined using a Probabilistic Risk Assessment 
(PRA). A PRA determines the overall risk associated 
with a particular mission by factoring in all known risks 
to the spacecraft during its mission. The threat to 
mission and human life posed by the micro-meteoroid 
and orbital debris (MMOD) environment is one of the 
risks. NASA uses the BUMPER II program to provide 
point estimate predictions of MMOD risk for the Space 
Shuttle and the ISS. However, BUMPER II does not 
provide uncertainty bounds or confidence intervals for 
its predictions. In this paper, we present possible 
approaches through which uncertainty bounds can be 
developed for the various damage prediction and 
ballistic limit equations encoded within the Shuttle and 
Station versions of BUMPER II. 

1. INTRODUCTION 

Micro-meteoroids and orbital debris (MMOD) are a 
serious problem for all manned and unmanned 
spacecraft. Micro-meteoroids are naturally occurring 
particles that are created from the breakup of asteroids 
and comets. Orbital debris is human-generated and 
orbits the Earth. Because these particles are traveling at 
many kilometers per second, even very small particles 
can cause loss of critical spacecraft systems or crew (in 
the case of manned spacecraft). 

NASA uses the BUMPER II code to calculate the risk 
of MMOD impact causing critical damage for each 
Space Shuttle mission as well as the risk of MMOD 
penetration for the International Space Station (ISS). 
BUMPER II has also been used to identity ways of 
reducing MMOD risk to within established program risk 
levels through operations, shielding, or other means 
(Williamsen, 2003). Space Shuttle mission profiles and 
operations have often been directly affected by risk 
predictions based on BUMPER II (e.g. its predictions 
were essential in determining the proper positioning of 
the payload bay door on STS-73 to provide protection to 
some otherwise lightly protected pressurized tanks 
within the payload bay). 


2. THE BUMPER II CODE 

The original BUMPER code was developed by Boeing 
under contract to NASA/MSFC for use on the Space 
Station Freedom program in 1986. In 1991, it was 
updated to the BUMPER II code, and configuration 
control was established at NASA/JSC in 1994. 
BUMPER II is now clearly considered the standard by 
which other MMOD risk assessment tools are 
measured — even the European and Russian Space 
Agencies have used versions of it. Currently, two 
versions of BUMPER II are maintained, one for the ISS 
program and one for the Space Shuttle program. The 
primary differences between the two versions are in the 
ballistic damage subroutines due to different exterior 
materials and failure criteria. 

While BUMPER II is a powerful tool, it does have 
limitations. BUMPER II results provide a point 
estimate of MMOD risk with no assessment of its 
associated uncertainty. Reporting risk predictions with 
uncertainty bounds enables those performing the 
program’s probabilistic risk assessments (PRAs) to fold 
the results into those assessments and put them in 
perspective with the other risk contributors. Risk 
predictions can also be used to help prioritize research 
programs to reduce the highest contributors to risk and 
uncertainty first. However, the uncertainties associated 
with underlying BUMPER II input models are still 
largely unknown. 

In this paper, we present and discuss possible 
approaches through which uncertainty bounds and/or 
confidence intervals can be developed for the various 
damage prediction (DP) and ballistic limit (BL) 
equations encoded within the shuttle and station 
versions of BUMPER II. The uncertainties in the 
code’s environment models and failure criteria 
definitions are beyond the scope of this study. 

3. NASA PROBABILISTIC RISK ASSESSMENT 
(PRA) FOR SHUTTLE AND STATION 

NASA currently maintains PRAs for the Space Shuttle 
and the ISS programs. A PRA attempts to determine the 
overall risk associated with a particular mission by 
factoring in all known risks (and their corresponding 



uncertainties, if known) to the spacecraft during the 
mission in the analysis. The threat to mission and 
human life posed by the MMOD environment is one of 
the risks. 

The primary PRA development application for both the 
Space Shuttle Program (SSP) and the International 
Space Station (ISS) is the fault-tree analysis code called 
SAPHIRE (Systems Analysis Program for Hands-on 
Integrated REIiability). BUMPER II data are provided 
to the appropriate PRA model development team (either 
SSP or ISS) as point values without associated 
uncertainty distributions. These furnished point values 
typically represent probabilities over a given mission for 
the Shuttle and for ten-year on-orbit periods of 
consideration for the ISS. The next two subsections 
present a brief overview of the current PRA processes 
for the shuttle and for the space station. 

3.1. Shuttle Probabilistic Risk Assessment 

The current Shuttle PRA is the first Space Shuttle 
Program (SSP) sponsored PRA of the Shuttle, and is a 
comprehensive, integrated PRA from launch to wheel 
stop on the runway. It also represents the first Shuttle 
PRA to include MMOD as a risk contributor. The 
primary figure of merit at this time is Loss of Crew and 
Vehicle (LOCV). Additional figures of merit may be 
warranted as the Shuttle PRA scope is increased to 
include ascent abort scenarios, additional on-orbit 
activities, and safe haven to ISS as a result of possible 
ascent debris damage. 

The Shuttle PRA addresses the five major elements of 
the Shuttle (Orbiter, SSMEs, SRBs, RSRMs, and ET), 
the three major phases of a typical mission (ascent, 
orbit, descent), and the major risk categories (functional 
failures, crew actions, external events, 
phenomenological events, and common cause events). 

It is an integrated model that accounts for dependencies 
between equipment, systems, mission phases, 
environment, and humans. The current Shuttle PRA 
attempts to take into account all reasonable contributors 
to risk, and then calculates the probabilities of likely 
combinations of equipment and systems failures that 
can lead to LOCV (NASA/JSC, 2002). NASA attempts 
to take into account the uncertainties in the DP/BL 
equations, the MMOD environment models, and the 
failure criteria by assuming the BUMPER II point 
estimate predictions are mean values combined with an 
assumed “error factor” to generate the corresponding 5 th 
and 95 th percentile values. 

3.2. Station Probabilistic Risk Assessment 

In order to incorporate the as-received data into the 
SAPHIRE PRA model, BUMPER II data are post- 


processed using an Excel spreadsheet and @RISK, a 
third-party commercial product used in conjunction with 
Excel. The Bumper II data used in this fashion is 
supplemented by data from MSCSurv, a spacecraft 
Monte Carlo-based survivability program developed by 
NASA (see, e.g., Vitali, 2003). Probability uncertainty 
distributions and conversions to appropriate ISS mission 
times are generated using Excel/@RISK, and the post- 
processed data are subsequently incorporated into 
SAPHIRE for MMOD penetration scenarios leading to 
loss of science, crew evacuation, and loss of module 
critical PRA end-states. The results of a previous 
independent expert panel review of the ISS PRA model 
included very strong recommendations for regular 
coordination and integration activities to be conducted 
between the PRA developers and in-line operational and 
system engineering experts (IPRP, 2002). 

4. REVIEW OF NASA SENSITIVITY ANALYSIS 

In an attempt to begin to quantify the uncertainties 
associated with BUMPER II predictions (with particular 
emphasis on shuttle analyses), NASA performed a 
sensitivity study of the dependence of BUMPER II risk 
prediction on five (5) key parameters using the 
BUMPER II code (Christiansen, et al, 2004). The code 
was initially run with baseline assumptions for a Space 
Shuttle mission. Then additional rims were performed 
varying only one of the baseline parameters by allowing 
it to take on either a predetermined low or a 
predetermined high value. The parameters selected for 
variation were those whose uncertainties were expected 
to have the greatest effect on MMOD risk predictions. 
However, other parameters that were known to have a 
large potential effect on MMOD penetration — such as 
MMOD particle shape — were omitted due to a lack of 
information on how, e.g., MMOD particle shapes might 
vary on orbit. As such, some of the potentially largest 
MMOD risk-producing factors still remain to be 
examined for their effects. Tabs. 1 and 2 show the 
parameters selected and, where appropriate, the high 
and low factor or value used in the sensitivity study. 

Table 1. Orbital Debris Sensitivity Study Variables 


(Christiansen, et al, 2004) 


Study 

Parameter 

Higher Risk 
Value 

Baseline 

Value 

Lower Risk 
Value 

Density 

(gm/cu.cm.) 

7.9 (steel) 

2.8 

(aluminum) 

1.0 

(water/ice) 

Flux Factor 

1.6 

1.0 

0.9 

BL Eqn 
Prediction 
Mult Factor 

0.8 

1.0 

1.2 

Failure 

Criteria 

Minimum 

Failure 

Criteria 

Baseline 

Failure 

Criteria 

Maximum 

Failure 

Criteria 

Velocity 

Distribution 

Maximum 

Velocity 

Option 

Baseline 

Velocity 

Option 

Minimum 

Velocity 

Option 




Table 2. Meteoroid Sensitivity Study Variables 
(Christiansen, et al, 2004) 


Study 

Parameter 

Higher Risk 
Value 

Baseline 

Value 

Lower Risk 
Value 

Density 

(gm/cu.cm.) 

1.9 

1.0 

0.5 

Flux Factor 

2.0 

1.0 

0.5 

BL Eqn 
Prediction 
Mult Factor 

0.8 

1.0 

1.2 

Failure 

Criteria 

Minimum 
Failure Criteria 

Baseline 

Failure 

Criteria 

Maximum 

Failure 

Criteria 

Velocity 

Distribution 

Maximum 
Velocity 
Option (Long 
Period Comets) 

Baseline 

Velocity 

Option 

Minimum 

Velocity 

Option 

(Asteroidal) 


The results of the sensitivity assessment are shown in 
Tab. 3. These results do not include the velocity 
parameter variation runs because the meteoroid runs had 
not yet been performed at the time of the completion of 
the study. The dramatic swings in the predicted MMOD 
risks shown in Tab. 3 demonstrate the importance of 


developing the capability to include overall uncertainty 
bounds as part of the results of MMOD risk assessments 
performed in support of future shuttle missions. 

5. BUMPER II UNCERTAINTY ANALYSIS ~ 
SHUTTLE VERSION 

The shuttle version of BUMPER II uses a variety of 
equations to predict damage to shuttle components in 
terms of an impacting particle’s density, velocity, and 
angle of impact. Some equations are developed by 
simply drawing a curve through fail/no-fail test data (the 
so-called ballistic limit, or BL, equations), while others 
are developed by performing statistical curve-fits to 
empirical data (the damage predictor, or DP, equations). 
Considering the different approaches used to derive 
them, the DP equations and BL equations in the shuttle 
version of BUMPER II belong to two different classes 
of empirical equations. These equations and their 
uncertainty considerations are discussed in this section. 


Table 3. Sensitivity Study Results (Christiansen, et al, 2004) 


Case 

Expected Number of 
Orbital Debris (OD) and 
Meteoroid (M) Penetrations (N) 

Probability of 
No Penetration 

MMOD 

Penetration 

% 

Change 

OD 

M 

MMOD Total 

MMOD Total 

Risk 

Odds 

From 
Baseline N 

m 

Baseline 

0.00215 

0.00205 

0.00420 

0.99581 

0.419% 

1 in 239 


i 

Max Flux 
Min Flux 


0.00410 

0.00102 

0.00754 

0.00296 


0.751% 

0.296% 

1 in 133 
1 in 338 

■ 

- | 

2 

Max Vel 
Min Vel 








3 

Max Den 
Min Den 

0.00954 

0.00046 

0.00518 

0.00090 

0.01471 

0.00136 

0.98539 

0.99864 

1.461% 

0.136% 

1 in 68 
1 in 734 

250% 

-68% 

4 

Max Eqn 
Min Eqn 

0.00103 

0.00466 

0.00102 

0.00457 

0.00204 

0.00923 

0.99796 

0.99081 

0.204% 

0.919% 

1 in 490 
1 in 109 

-51% 

120% 

5 

Max Fail 
Min Fail 

0.00087 

0.01224 

0.00091 

0.01198 

0.00178 

0.02422 

0.99822 

0.97607 

0.178% 

2.393% 

1 in 561 
1 in 42 

-58% 

477% 


5.1. Damage Predictor Equations 

The DP equations are curve-fits to empirical data, that is, 
they are the results of statistical regression analyses of 
available test data. As such, uncertainty bounds and/or 
confidence intervals can be obtained at the time that the 
regression analyses are being performed to form the DP 
equations. A review of the literature referencing the 
various DP equations in BUMPER II revealed that while 
an R 2 value for a DP equation may have been 
occasionally provided, information regarding uncertainty 
bounds and/or confidence intervals was not (R 2 values 
can be used as a metric to quantify how well a regressed 
curve fits the data on which it is based). Considering the 
sophistication of modem statistical analyses packages 
and the availability of the test data in electronic format, 
obtaining this information would be a relatively 


straightforward task that would involve redoing the 
original regression analyses. 

5.2. Ballistic Limit Equations 

Unlike the DP equations, the BL equations are not 
statistically based. They are not curve-fits, but are rather 
simply lines of demarcation between regions of 
penetration and non-penetration. As a result, and also 
wnlike the DP equations, it is simply not possible to 
obtain uncertainty bounds and/or confidence intervals as 
part of the current procedure that is used to derive the 
BL equations. Alternative, innovative approaches must 
be developed to either (i) obtain the required uncertainty 
information from existing BL equations and the data on 
which they are based, or (ii) rederive the BL equations 
using a statistics-based approach so that uncertainty 














information is forthcoming out of the analyses along 
with the equations themselves. Both of these options are 
briefly discussed in the following paragraphs. 

(a) Uncertainty Modeling Using Existing BL 
Equations and Data 

As discussed previously, considering the manner in 
which the existing BL equations are derived, it is 
impossible to state that any existing BL equation is 
accurate to within +/-X% with a confidence of Y%. 
However, it may be possible to develop a quantitative 
measure that would indicate, at least at some level, the 
accuracy of a BL in separating the region of perforating 
projectile diameter-impact velocity combinations from 
non-perforating combinations. We can consider, for 
example, the use of specificity and sensitivity ratios, 
which are used in the medical world to distinguish 
between false positives and false negatives. For example, 
if we designate a penetration event as the event we are 
testing for, then a penetration might be considered as a 
“positive reading” and a non-penetration might be 
considered as a “negative reading”. In this case, the 
following definitions could be applied for any given BL 
equation: 

Sensitivity = (Actual penetrations predicted as 
penetrations) / (Actual penetrations predicted as 
penetrations + Actual penetrations predicted as non- 
penetrations) (1) 

Specificity = (Actual non-penetrations predicted as non- 
penetrations) / (Actual non-penetrations predicted as 
non-penetrations + Actual non-penetrations predicted as 
penetrations) (2) 

By using these ratios we would get a first-order 
quantitative assessment of whether or not a given BL 
equation tends to be conservative or non-conservative (at 
least in the tested areas). For example, a low specificity 
value (i.e. more non-penetrations predicted as 
penetrations) and a high sensitivity value (i.e. fewer 
penetrations predicted as non-penetrations) would tend 
to demonstrate a conservatism in the BL equation 
whereas a high specificity and a low sensitivity would 
tend to demonstrate non-conservatism. If both values 
were relatively high, that would indicate a fairly accurate 
curve, whereas if both values were fairly low, that might 
demonstrate a problem with the testing method or with 
test repeatability. 

As an example, consider the BUMPER II subroutine 
TPSBELPERF, which calculates the critical particle 
diameters that will cause threshold perforation of an 
underlying 2024-T81 aluminum plate or honeycomb 
sandwich panel using the BL equations given in 
Christiansen and Friesen (1997). The perforation/no- 
perforation data on which these equations are based can 


be found in Friesen and Whitney (1996). The data used 
to generate the BL equations in this subroutine were 
obtained from tests on five (5) different tile 
configurations using spherical steel and aluminum 
projectiles fired at varying trajectory obliquities. 

The vast majority of the tests fired used aluminum 
projectiles and target configuration consisting of a 
ceramic tile backed by a thin aluminum plate. When the 
data for this particular TPS target type are plotted against 
the corresponding BL equation (Christiansen and 
Friesen, 1997), the correlations appear to be very 
favorable (see Fig. 1). However, for other target types, 
with which significantly fewer tests were performed, the 
comparisons were not always as favorable. 



Impact Velocity (km/s) 


Figure 1. Comparison of Ballistic Limit Curves and 
Test Data, Basic TPS Target Type 

Tab. 4 below shows a summary of how correct (C) or 
incorrect (IC) the TPSBELPERF BL equation is in 
predicting penetration (P) or non-penetration (NP) 
events over the limited velocity ranges at which the tests 
were performed. 


Table 4. Overview of Penetration Event Prediction 
for Subroutine TPSBELPERF 


Targ 




BUf 

Kc-358 


# 


nil 

ESI 

K*n 

RSI 

1 

Steel 

0 

3 

2 

2 

0 

1 

Aluminum 

0 

7 

0 

3 

0 

1 

Aluminum 

30 

2 

0 

1 

1 

1 

Aluminum 

60 

4 

0 

5 

0 

2 

Aluminum 

0 

2 

0 

0 

1 

3 

Aluminum 

0 

1 

3 

0 

2 

4 

Aluminum 

0 

3 

0 

4 

1 

4 

Aluminum 

60 

n/a 

n/a 

3 

0 

5 

Aluminum 


1 



2 


Tab. 5 presents the specificity and sensitivity ratios for 
the entire dataset used to develop this BL equation as 
well as for some subsets of it. As can be seen from the 
relatively high overall ratio values in Tab. 5, overall, the 
TPSBELPERF BL equation is able to discriminate fairly 
well between penetration and non-penetration events. If 





















































only steel projectiles are considered, the relative values 
of the specificity and sensitivity ratios indicate a non- 
conservatism in the BL equation when used for impact 
scenarios involving steel projectiles. 


Table 5. Sensitivity and Specificity Ratios 



Sensitivity 

Ratio 

Specificity 

Ratio 

Overall 

82.14% 

72.00% 

ST PRO J 

60.00% 

100.00% 

AL PROJ 

86.96% 

69.57% 

AL, 0-deg 

82.35% 

53.85% 

AL, non-O-deg 

100.00% 

90.00% 


However, if only aluminum projectiles are considered, 
the relative values of the specificity and sensitivity ratios 
indicate a conservatism in the BL equation. This 
conservatism for aluminum projectiles is seen again by 
the relative values of these ratios in the 0-deg dataset as 
well. While some level of conservatism is appropriate, 
especially for mission systems that support the presence 
of humans in space, an exceedingly high amount of 
conservatism may actually be detrimental to other 
mission aspects. If BUMPER II results are being 
represented as the best point estimates of MMOD risk at 
this time, an effort should be made to eliminate any 
unreasonable conservatism where it might exist. 

(b) A Statistics-Based Approach to Uncertainty 
Modeling for BL Equations 

The approach presented in the preceding section 
produces a result that, at best, really only measures, 
albeit in a quantitative fashion, the scatter (or 
repeatability) of the tests or, perhaps, the reliability of 
the BL equation based on the tests. It does not provide 
enough information to allow one to make the statement 
that a given BL equation is accurate to within +/-X% 
with a confidence of Y%. In order to be able to make 
this statement, the BL equations must be rederived using 
a consistent, statistics-based approach. Such an approach 
was proposed and used by Williamsen and Jolly (1992) 
to develop preliminary BL curves for the Space Station 
Freedom manned module multi-wall orbital debris 
shields. 

In Williamsen and Jolly (1992), data were regressed to 
develop an empirical equation that defined a 
penetration parameter P m in terms of impact 
parameters for a given set of target material properties 
and geometry, that is, 

p m = Ad p ,v p ,e p )=ad f p v; cos* e p +s p> 

In eqn (3), a through s are coefficients obtained through 
a standard nonlinear regression of P m data. In its simplest 
form P m may be visualized as a measure of the depth of 
penetration through an entire multi-wall shield system. It 


includes crater depth data prior to perforation of a 
critical target region as well as witness plate data after 
the perforation of a critical target region. 

In the context of a multi-wall shield, if the impact event 
results in a perforation of the bumper, but no penetration 
of the pressure wall, P m =0. If the pressure wall is 
penetrated, then P m = t w , the thickness of the pressure 
wall. If the first witness plate is also perforated, then 
P m = t w + t wp i, where t wpl is the thickness of the first 
witness place; if the second is perforated, P m = t w + t wp i + 
t wp2 ; etc. If the pressure wall is cratered, but not 
perforated, then 0<P m =d c <t w , where d c is the depth of the 
deepest pressure wall crater. Setting the penetration 
parameter equal to a predetermined value (i.e. t w ) 
allowed Williamsen and Jolly (1992) to solve for critical 
diameter in terms of impact velocity that would result in 
just barely perforating the pressure wall. That is, using a 
statistics-based approach, the authors were able to arrive 
at a BL equation for a variety of multi-wall systems. It is 
important to note that this approach could be easily 
modified to include rear side spall considerations in the 
calculation of P m if appropriate for the target type being 
considered. 

If this approach were to be adapted to shuttle TPS 
configurations, two important results would follow. 
First, we would be able to develop statistics-based (and 
not simply hand-drawn) BL equations for shuttle TPS 
configurations. Second, we would be able to obtain, for 
each BL equation so derived, the statistics-based 
uncertainty information that would allow us to make the 
statement that a given BL equation is accurate to within 
+/-X% with a confidence of Y%. Since this is the type of 
information that is needed to develop overall uncertainty 
bounds for MMOD predictions, it would appear that this 
approach is the appropriate one to take. 

6. BUMPER II UNCERTAINTY ANALYSIS - 
STATION VERSION 

The subroutines in the station version of BUMPER II 
consist exclusively of BL equations. These equations are 
again simply lines of demarcation between regions of 
penetration and non-penetration and are not statistically 
based. As such, considering the manner in which the 
existing station BL equations were derived, it is 
impossible to state that any existing BL equation is 
accurate to within +/-X% with a confidence of Y%. 
Again, alternative, innovative approaches must be 
developed to either (i) obtain the required uncertainty 
information from existing BL equations and the data on 
which they are based, or (ii) rederive the BL equations 
using a statistics-based approach so that uncertainty 
information is forthcoming out of the analyses along 
with the equations themselves. Based on the discussions 
in subsections (a) and (b), it would appear that the latter 




is the preferred option. In addition, a similar approach 
should be followed in determining uncertainty bounds 
and/or confidence intervals for the ballistic limit curves 
beyond the testable regime. 

7. CURRENT NASA PLANS FOR FUTURE 
BUMPER II UNCERTAINTY ANALYSIS 

As BUMPER II is the accepted code for MMOD risk 
assessments for the Space Shuttle and the ISS, NASA 
also plans to use BUMPER II as the engine for running 
Monte Carlo type analyses (which form the backbone of 
current PRAs). BUMPER II is generally run using script 
files which basically provide the responses an interactive 
user would otherwise provide. In order to perform a 
Monte Carlo style analysis, many script file commands 
would have to be in place in order to run through the 
Response and Shield modules of BUMPER II many 
times. A preprocessor is currently being developed to 
create the necessary script file(s) to run these many 
iterations of BUMPER II. Within the script file(s), the 
iterations would contain parameter values which would 
be varied from iteration to iteration based on random 
numbers drawn against user-specified distributions. 

After completion of the BUMPER II runs, the 
postprocessor to BUMPER II (also being developed) 
will be run to read expected failure numbers from the 
shield summary files that BUMPER II provides for each 
iteration, calculate the mean and user-specified high and 
low percentile values, and write the results for all 
iterations along with the mean, and high and low 
percentile values to a single, Excel-compatible file. The 
postprocessor will also calculate certain key 
convergence statistics to allow the user to assess whether 
convergence has been achieved. Note: the postprocessor 
will be designed to run independently of BUMPER II, so 
this is not a stop-when-convergence-is-met capability. 

8. CONCLUSIONS 

The BUMPER II code is currently used by NASA and 
other space agencies to calculate the risk of MMOD 
impact causing critical damage for each Space Shuttle 
mission, and the risk of MMOD penetration for the 
International Space Station (ISS). While BUMPER II is 
a powerful tool, it does have limitations - it provides a 
point estimate of MMOD risk, but without any 
assessment of its associated uncertainty. In this paper, 
we have presented several possible approaches through 
which uncertainty bounds and/or confidence intervals 
can be developed for the various damage prediction (DP) 
and ballistic limit (BL) equations encoded within the 
shuttle and station versions of BUMPER II. 


accuracy of a BL in separating the region of perforating 
projectile diameter-impact velocity combinations from 
non-perforating combinations, these ratios, at best really 
only measure the scatter (or repeatability) of the tests or, 
perhaps, the reliability of the BL equation based on the 
tests. To determine actual confidence bounds, the BL 
equations in BUMPER II must be rederived using the 
consistent, statistics-based approach described herein. 
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